Measures of success: characterizing teaching and teaching change with segmented and holistic observation data

Abstract Background Numerous studies show that active and engaging classrooms help students learn persist in college, but adoption of new teaching practices has been slow. Professional development programs encourage instructors to implement methods change the status quo STEM undergraduate teaching, structured observations can be used multiple ways describe assess this instruction. We addressed challenge measuring instructional with observational protocols, data often do not lend themselves easily statistical comparisons. Challenges using comparative research designs include lack descriptive utility for holistic measures problems related construct representation, non-normal distributions Type-I error inflation segmented measures. Results grouped 790 mathematics classes from 74 Latent Profile Analysis (a clustering technique) found four reliable categories classes. Based on grouping we proposed a simple proportional measure called Proportion Non-Didactic Lecture (PND). The aggregated proportions interactive lecture each instructor. tested PND derived Reformed Teaching Observation Protocol (RTOP) professional study. worked hypothesis tests lacked some power due possible ceiling effects. However, provided effective descriptions changes approaches pre post. In tandem examining measure, also examined RTOP-Sum, an existing outcome comparison studies. is based items protocol. As aggregate it highly reliable, correlated PND, had more than PND. RTOP did provide thick afforded by Conclusions Findings suggest useful dependent both Both have strengths weaknesses: are best at describing while power. Determining validity these important future use